1 research outputs found
The Effects of Computational Resources on Flaky Tests
Flaky tests are tests that nondeterministically pass and fail in unchanged
code. These tests can be detrimental to developers' productivity. Particularly
when tests run in continuous integration environments, the tests may be
competing for access to limited computational resources (CPUs, memory etc.),
and we hypothesize that resource (in)availability may be a significant factor
in the failure rate of flaky tests. We present the first assessment of the
impact that computational resources have on flaky tests, including a total of
52 projects written in Java, JavaScript and Python, and 27 different resource
configurations. Using a rigorous statistical methodology, we determine which
tests are RAFT (Resource-Affected Flaky Tests). We find that 46.5% of the flaky
tests in our dataset are RAFT, indicating that a substantial proportion of
flaky-test failures can be avoided by adjusting the resources available when
running tests. We report RAFTs and configurations to avoid them to developers,
and received interest to either fix the RAFTs or to improve the specifications
of the projects so that tests would be run only in configurations that are
unlikely to encounter RAFT failures. Our results also have implications for
researchers attempting to detect flaky tests, e.g., reducing the resources
available when running tests is a cost-effective approach to detect more flaky
failures.Comment: This work has been submitted to the IEEE for possible publication.
Copyright may be transferred without notice, after which this version may no
longer be accessibl